Device and Method for Superimposing Patterns on Images in Real-Time, Particularly for Guiding by Localisation

ABSTRACT

The invention relates to a device for superimposing known patterns, characteristic of a region, on (real) images of said region. The device comprises, a memory in which patterns are stored, which are representative of a selected region, of known position and orientation with relation to a common reference and processing means, for determining a pattern representative of the selected portion in the memory, on receipt of the designation of at least one portion of an observed image of the selected region, taken at a selected angle and at least one representative attribute of said region, taking account of the attribute selected, then superimposing the determined pattern on the selected portion of the image taking account of the selected angle.

The invention relates to the field of image data processing, and moreprecisely to the real-time registration of data images representingknown patterns on observation images.

In a number of fields, it is important to know precisely, at eachinstant, how one is positioned relative to a location or an object, ormore generally a region.

This is the case for example in the field of surgery, particularly whenit involves the so-called “mini-invasive” technique. In this case, theoperating field is observed by an endoscopic camera introduced into thepatient's body and delivering images to one or more monitors (orobservation spectacles) at which a surgeon is positioned. In the case ofa robot-assisted procedure, the surgeon remotely controls the robotmanipulator arms of which the ends are also introduced into thepatient's body. This applies in particular is to the Da Vinci®installation of Intuitive Surgical Inc which includes, on one hand, astereoscopic display device, giving the surgeon a three-dimensional (3D)view of the operating region, and on the other hand, manual controlhandles and pedals enabling the surgeon to address handling instructionsand commands to the robot.

This operating technique is particularly beneficial for the patient inthat it is minimally invasive. However, it is particularly awkward toput into effect by the fact that it only offers the surgeon, on onehand, a partial and to some extent distorted view of the region in whichhe/she is required to operate, due to the utilization of an endoscopiccamera, and on the other hand, a very confined operating space that isencumbered by the robot manipulator arms and the endoscopic camera.Moreover, given that certain regions, such as the heart, are animated,the difficulty of the intervention is thereby accentuated.

To improve the situation, it has been proposed to implement apre-operative modeling phase. In such a preoperative phase, athree-dimensional, and possibly temporal, model of the region to beoperated on is constructed using images obtained by medical imaging. Inthe case of a heart, the coronary network is also determined with theaid of angiographic sequences, then this coronary network is overlaidonto the surface of the heart obtained by MRI. An anatomical model ofthe part of the patient's body containing the region to be operated onis then created, again using images obtained by medical imaging.

The optimal incision points are then determined, on one hand, takinginto account the anatomical model and parameters such as the dexterityand the accessibility of the target region, and on the other hand, theoptimal configuration of the robot arms, so as to avoid collisions andto obtain maximum separation, in particular.

On completion of the pre-operative phase, the surgeon can proceed withthe operation. The patient is then placed on the operating table, thenthe endoscope is calibrated using a grid placed on the operating tableand observed from different points of view. The patient's body is thenincised at the optimal incision points previously determined. The robotarms are then positioned in the optimal configuration previouslydetermined, and their ends, together with the tip of the endoscopiccamera, are introduced into the patient's body via the incisions. Theoperation can then begin.

Despite the pre-operative phase, the surgeon may still experiencedifficulty in precisely locating the intervention area. This can occurparticularly in the case of an intervention on an organ such as theheart. It may in effect be difficult to locate the interventricularartery due to an excess of fat at the surface of the epicardium. It isalso possible to confuse the marginal branch of the circumflex artery orthe diagonal branch (particularly developed) with the interventricularartery due to the high magnification of the endoscopic camera and/or thesmall field of view and/or the limited available perspective and/or poorpositioning of the opening made in the pericardium.

Added to these difficulties in locating the operating targets, thesurgeon can also encounter difficulties in positioning the endoscopiccamera and therefore in estimating the angle at which the operatingregion is observed. Furthermore, given the lack of tactile feedback, itis not possible to identify the area to be operated on by pressing on itwith the ends of the robot arms.

To sum up, the surgeon can experience real difficulty in determining theprecise position of the area (or portion) of the observed region inwhich he/she must operate, relative to the known positions of the endsof the surgical robot arms.

To further improve the situation, it has been proposed to assist thesurgeon by superimposing on the endoscope images of the observed regiona pattern representing an element characteristic of the portion of theregion where the operation has to be carried out, or an adjacentportion. These patterns are pre-extracted from digital models generatedfrom images obtained by medical imaging. However, such assistance is notsufficient, particularly when the intervention region includes severalsubstantially identical characteristic elements (either naturally, or byvirtue of the observation angle), as in the case of the coronarynetwork.

Similar difficulties in pinpointing the precise position of locations orobjects, via portions of real images, also arise in other technicalfields, and particularly in the field of urban guidance.

The object of the invention is therefore to remedy all or some of theaforementioned drawbacks.

To this end a device is proposed for superimposing known patterns (forexample three-dimensional (3D) patterns), characteristic of a region, on(real) images of this region in real time.

This device is characterized in that it includes, on one hand, a memoryin which are stored patterns representing portions of a selected regionand of known position and orientation relative to a common reference,and on the other hand, processing means for determining a patternrepresentative of the designated portion in the memory, on receipt ofthe designation of at least one portion of an observed image of theselected region, taken at a selected angle, and at least onerepresentative attribute of this portion, taking account of thedesignated attribute, then superimposing the determined pattern on thedesignated image portion taking account of the selected angle.

The device according to the invention may include other characteristicswhich may be taken separately or in combination, and particularly:

-   -   processing means capable of performing the registration by        successively designating portions of observation images of the        selected region and/or attributes representing these portions,    -   a memory capable of storing a three-dimensional model        representing the region within the selected reference. In this        case, the processing means are preferably designed to perform        the registration(s) in particular using the stored        three-dimensional model,    -   processing means capable of determining at least one measurement        equation from the designation of an image portion, an attribute        and at least one hypothesis, and of determining the pattern and        its registration as a function of the determined measurement        equation. Several measurement equations can be determined from        the designation of several image portions, attributes and        hypotheses. In this case, the processing means determine the        pattern and its registration as a function of a combination of        the determined measurement equations. The registration is of the        rigid type, for example, by minimization of a selected        criterion, taking account of the measurement equations derived        from the hypotheses. Furthermore, the hypotheses can be        transmitted to the processing means by the user making the        designations, or can be directly determined by the processing        means,    -   processing means capable of supplying image data representative        of a registered pattern so that it can be observed at the same        time as the observation images, in superimposed fashion, on the        corresponding designated image portion, once the registration        has been done,    -   a memory capable of storing a correspondence table between the        patterns and information data representing them. In this case,        the processing means are advantageously designed to provide,        whenever they are instructed to do so, information data        representing a registered pattern,    -   processing means capable of providing, substantially at the same        time and whenever they are instructed to do so, image data        representing a registered pattern and the information data        representing this pattern.

The invention also relates to an installation for guidance by locationincluding, in a first part, observation means capable of providingimages of a selected region at a selected angle, in a second part, adisplay device (for example one or more computer monitors or observationspectacles) enabling these images to be displayed, in a third part, aman/machine interface enabling a user to designate at least one portionof the region represented by the displayed images and at least oneattribute representing this portion, and in a fourth part, aregistration device of the type described above, furnished withportion(s) and attribute(s) designated by the man/machine interface anddelivering image data representing a registered pattern so that thelatter is superimposed by the display device on the designated portiondisplayed, once the registration has been performed.

The observation means may include acquisition means, for example of theendoscope type, the position of which is known at each instant relativeto a calibration reference, from which the position of the observedregion is defined, and capable of delivering observation images to thedisplay device.

When the installation is used for guidance only, for example urbanguidance, it preferably includes control means designed, when theyreceive a request designating a pattern of the observed region, toinstruct the registration device to determine position data representingthe position of this pattern in relation to the calibration reference,taking account of the registration, then of specifying controlinstructions intended to guide the user to the portion matching thispattern.

When the installation is used for surgical interventions, it can includea surgical robot incorporating arms whose respective positions inrelation to the calibration reference are known at each instant andwhich can be remotely controlled by instructions provided by a user viathe man/machine interface. However, it also includes control meanslinked to the registration device and to the man/machine interface, anddesigned, when they receive a request designating a pattern of theobserved region, on one hand, to instruct the registration device todetermine position data representing the position of this pattern inrelation to the calibration reference taking account of theregistration, and on the other hand, to specify control instructionsintended to move the robot arms in the vicinity of the portion of theregion matching the designated pattern.

The invention also relates to a process for real-time registration ofknown patterns (for example 3D), characteristic of a region, onto imagesof this region, and consisting of:

-   -   storing in a memory patterns representing portions of a selected        region and of known position and orientation relative to a        common reference,    -   observing the selected region at a selected angle and delivering        observation images of this region in real time,    -   designating at least one portion of the observation image of the        selected region and at least one attribute representing this        portion,    -   identifying in the memory a pattern representing the designated        portion, taking into account the designated attribute, and    -   overlaying the pattern on the designated image portion taking        the selected angle into account.

The method according to the invention may include other characteristicswhich may be taken separately or in combination, and particularly:

-   -   the registration may be performed by successively designating        portions of observation images of the selected region and/or        attributes representing these portions,    -   a 3D model representing the region in the selected reference may        be stored in the memory, and each registration may be performed        at least from this 3D model,    -   a rigid type registration may be performed, by minimization of a        selected criterion. In this case, at least one measurement        equation is preferably determined from the designation of an        image portion, an attribute and at least one hypothesis, and the        pattern is determined and the rigid registration is performed as        a function of the determined measurement equation. Several        measurement equations can be determined from the designation of        several image portions, attributes and hypotheses, constituting        the constraints, and the pattern can be determined and its rigid        registration performed as a function of a combination of the        determined measurement equations. At least some of the        hypotheses can be transmitted by the user making the        designations, for example in the form of attributes,    -   image data representing a registered pattern can be provided so        that it is observed at the same time as the observation images,        and in a superimposed fashion, on the corresponding designated        image portion,    -   a correspondence table between the patterns and information data        representing these patterns can be stored in the memory, and        information data representing a pattern can be delivered on        request,    -   image data representing a registered pattern can be displayed,    -   the designations can be performed via a man/machine interface,    -   when a request is received from a user designating a pattern        contained in the observed region, position data representing the        position of this pattern can be determined in relation to the        calibration reference (from which the position of the observed        region is defined), taking the registration into account, and        control instructions intended to guide the user to the portion        corresponding to said designated pattern can then be determined,    -   provision can be made for a surgical robot incorporating arms        the respective positions of which in relation to the selected        reference are known at each instant and can be remotely        controlled by instructions supplied by a user via the        man/machine interface, and when a user request is received        designating a pattern of the selected region, position data        representing the position of the pattern can be determined in        relation to the calibration reference, taking the registration        into account, and control instructions intended to move the        robot arms in the vicinity of the portion of the region matching        the designated pattern can, then be determined.

Other features and advantages of the invention will become apparent uponexamination of the following detailed description together with theattached drawings in which:

FIG. 1 is a diagrammatic illustration of an embodiment of aninstallation according to the invention suitable for application in thefield of mini-invasive telesurgery,

FIG. 2 is a diagrammatic illustration of a 3D pattern representing acoronary artery,

FIG. 3 is a diagrammatic illustration of a 3D pattern representing threeadjacent coronary arteries,

FIG. 4 is a diagrammatic illustration of a 3D pattern representing acoronary artery and three bifurcations,

FIG. 5 is a diagrammatic illustration of a 3D pattern representing anangular sector at a bifurcation between two coronary arteries, and

FIG. 6 is an image of a heart onto which a 3D model has beensuperimposed, after registration.

The attached drawings may not only serve to complement the invention,but may also contribute to its definition, as the case may be.

The invention relates in a general manner to real-time registration ofimage data representing known patterns, for example three-dimensionalpatterns (3D), characteristic of a region, onto observation images ofthis region. However, it also relates to location guidance installationsusing registration of this kind, such as for example urban guidance andtelesurgery systems, particularly of the “mini-invasive” type.

Reference will first be made to FIG. 1 in describing a non-restrictiveembodiment of an installation according to the invention suitable formini-invasive telesurgery.

The telesurgery installation illustrated is for example composed of theDa Vinci® installation of Intuitive Surgical Inc. Schematically, itincludes a control station CS including a chair 1 enabling a surgeon Sto sit at a console equipped with a control keyboard (not shown), afirst manual control 2 for the left hand, a second manual control 3 forthe right hand, a display device 4, in this instance of the stereoscopictype, and a set of control pedals 5.

Each manual control 2, 3 includes for example a control lever 6, 7 (ofthe “joystick” type) intended to control one of the manipulator arms 8,9 of a robot 10, which will be described below, and the manipulator arm11 of a stereoscopic camera 12, which will also be described below, andone or more control keys 13, 14 (of the touch-sensitive type, orpushbutton type, or “mouse” type).

The set of pedals 5 includes for example a pedal enabling a manualcontrol 2, 3 to be assigned to control the surgical robot 10, a pedalenabling a manual control 2, 3 to be assigned to control the camera 12,and a pedal enabling a manual control 2, 3 to be assigned to control aninstallation control module 15, which will be described below.

The control keyboard, the manual controls 2 and 3, and the set ofcontrol pedals 5 constitute a man/machine interface.

The display device 4 includes, in this instance, a first screen 16, forthe display of real two-dimensional (2D) images delivered by a firstchannel of the camera 12 and intended for the left eye of the surgeon S,and a second screen 17, for the display of real two-dimensional (2D)images delivered by a second channel of the camera 12 and intended forthe right eye of the surgeon S.

The surgical robot 10 is intended to be placed in proximity to theoperating table 18, on which the patient P is placed for themini-invasive operation. It generally includes two manipulator arms 8and 9 equipped with ends adapted to the operation and intended to beintroduced into the body of the patient P via incisions.

The stereoscopic camera 12 includes a manipulator arm 11 the end ofwhich carries two endoscopic optical fibers 19 defining two imageacquisition channels.

The surgical robot 10 and the endoscopic camera 12 can be combined toform a “master robot”.

The installation additionally includes a control unit 20, for examplearranged in the form of a workstation, including the control module 15and a registration device according to the invention D, which will befurther described below. The control module 15 is linked to the consoleof the control station CS, to the surgical robot 10, to the stereoscopiccamera 12 and to the registration device D.

The registration device D, according to the invention, is intended in ageneral manner to overlay known patterns, which are characteristic of aregion (in this instance where an operation is to be carried out), onto(real) images of this region in real time. In the following description,it is considered that the patterns are three-dimensional (3D), but theycan be two-dimensional (2D), at least in some instances.

This device D firstly includes a memory 21 storing three-dimensional(3D) patterns representing characteristic portions of the region inwhich the operation is to take place, and of known position andorientation relative to a common reference (or pre-operative reference).The device D also includes a processing module 22 linked to the memory21 and designed, when it receives the designations, on one hand, of atleast one portion of an observation image of the intervention region,taken at a selected angle by the endoscopic camera 12 and delivered bythe control module 15, and on the other hand, of at least one attributerepresenting the designated portion, to determine in the memory 21 a 3Dpattern representing this designated portion, taking account of thedesignated attribute and the selected viewing angle, and then tosuperimpose the determined 3D pattern on the designated image portion.

It is important to note that a set of 3D patterns may constitute a 3Dmodel. Therefore, the registration may apply not only to a 3D pattern,but also to a 3D model.

In the case of the heart, for example, a 3D model may represent thecoronary tree, this 3D model then being composed of a multiplicity of 3Dpatterns representing structures characteristic of the coronary tree,such as arteries, junctions and bifurcations for example. In fact, inthe case of a coronary tree, two types of structures are defined. Afirst type includes curves representing the arteries, while a secondtype includes characteristic elements, such as junctions or bifurcationsfor example. These different types are illustrated in FIGS. 2 to 5.

More precisely, FIG. 2 illustrates a pattern representing a coronaryartery, FIG. 3 illustrates a pattern representing a configuration ofthree coronary arteries, FIG. 4 illustrates a pattern representing aconfiguration of three bifurcations on a coronary artery, and FIG. 5illustrates a pattern representing an angle α characteristic of abifurcation between two coronary arteries.

For example, a bifurcation is defined, in a first part, by an index(integer identifying the bifurcation), in a second part, by two indexes(Art1 and Art2 which identify the two arteries concerned), and in athird part, by a point (triplet of coordinates of the bifurcation in thepre-operative reference). Similarly, an artery is defined, on one hand,by an index (integer identifying the artery) and, on the other hand, bya set of parameters of a B-spline constituting the center line of theartery (quadruplet (xi, yi, zi, ui) in which (xi, yi, zi) is a tripletdefining each control point of the artery in the pre-operativereference, and ui denotes a node of the B-spline of which the value isbetween 0 and 1).

As the generation of 3D patterns is not the object of the invention, itwill not be described here in detail. A precise description of a mode ofgenerating these patterns is for example given in the paper by EveCoste-Manière et al “Optimal Planning of Robotically Assisted HeartSurgery: Transfer Precision in the Operating Room”, B. Sicilian and P.Dario (Eds.): Experimental Robotics VIII, STAR 5, pp. 424-434, 2003, orin the paper by Fabien Mourgues et al “3D+t Modeling of coronary arterytree from standard non simultaneous angiograms”: Proc. of MICCAI, Volume2208 of LNCS, Springer (2001), 1320-1322.

It is simply noted that the generation of 3D patterns of an interventionregion, such as for example an organ like the heart, first requiresthree-dimensional, and possibly temporal (3D+t), modeling of theintervention region from images obtained by medical imaging (MRI,scanner, etc). For example, in the case of a heart, a volumetric (3D)model of the heart can be obtained by MRI at a given instant in thecardiac cycle. To obtain a complete 3D+t model, the volumetric (3D)model is animated from a 3D+t model of the coronary network of themodeled heart, obtained from sequences of angiogram images (X rays),also referred to as coronarograms, taken at different angles and overseveral cycles, and synchronized relative to an electrocardiogram (ECG).

In the foregoing example of a 3D model of a coronary tree, the 3Dpatterns are therefore fractions (or portions) characteristic of the 3Dcoronary tree (or network) of which the positions are known in relationto the volumetric (3D) model of the heart of the patient P.

The pre-operative reference against which the positions of the 3Dpatterns and the heart are defined is normally that of the outerenvelope of the patient P. It is effectively in relation to this outerenvelope that the surgical robot 10 and the endoscopic camera 12 can becalibrated. Furthermore, the fact that the position of the heart, andtherefore its coronary network, is related to the outer envelope makesit possible to register the patient in relation to the robot in theoperating room.

A 3D anatomical model of the patient's body is also determined, at leastin an extensive part containing the region to be operated on, againbased on images obtained by medical imaging.

The positions, orientations and configurations of the 3D patterns (andthe models) are therefore stored in the memory 21 relative to a commonreference defined by means of index marks placed on the outer envelopeof the patient P.

Preferably, the 3D model of the heart of the patient P, and the 3Danatomical model, are also stored in the memory 21.

As the object of the invention does not include the pre-operativeplanning phase of the surgical operation, this will not be describedhere. Therefore, the following description deals with the application ofthe device D according to the invention, within an installationaccording to the invention, in the operating phase.

It will simply be noted that the pre-operative planning phase, which isoptional only, has the object in particular of determining the threeoptimal incision points that will allow the ends of the manipulator arms8, 9 and 11 of the surgical robot 10 and the endoscopic camera 12 to beinserted, taking into account the 3D anatomical model of the body of thepatient P, and parameters such as the dexterity and the accessibility ofthe target region. It also involves determining the optimalconfiguration of the manipulator arms 8 and 9 of the surgical robot 10to avoid collisions and to obtain maximum separation.

The device D operates within the installation once the arm 11 and thecamera 12 (endoscopic as the case may be) have been calibrated. Thisinvolves determining precisely the optical parameters of the camera andits position in the reference system of the arm 11. An example of amethod of endoscopic calibration is described in the paper by F.Mourgues et al “Flexible calibration of actuated stereoscopic endoscopefor overlay in robot assisted surgery”, Proc. of MICCAI, Volume 2488 ofLNCS, Springer (2002), 25-34. The calibration data are preferably storedin a memory of the control module 15.

Incisions are then made in the body of the patient P, at the optimalincision points determined during the preoperative planning phase. Themanipulator arms 8 and 9 of the surgical robot 10 and the arm 11 of theendoscopic camera are then placed in the optimal configuration, alsodetermined during the pre-operative planning phase, and their respectiveends are introduced into the body of the patient P via the incisions. Itis to be noted that the determination of the optimal incision points isa preferred option, but not obligatory, it being possible for thepositioning of the robot arms and the endoscope to be determinedempirically by the surgeon.

The two channels of the endoscopic camera 12 deliver their respectivesequences of 2D images to the control module 15, which transmits them tothe display device 4 so that they are displayed on the screens 16 and17. The surgeon S can then observe the region in which the object of theoperation, namely the heart H, is located, at the observation angle ofthe endoscopic camera 12. FIG. 6 is an image of the heart H of the typeobserved by the surgeon S on the screens 16 and 17.

Knowing the intervention region, it is possible at the start of theoperation to propose a first overlay of the 3D model (in this instancethe coronary tree) onto the displayed observation images. This can bedone manually or by external registration of the patient placed on theoperating table with the patient's pre-operative model. Externalregistration involves first using the robot end [sic] to point atseveral radio-opaque markers pre-attached to the patient's thorax, andpreviously segmented in the scanner images. The rigid transformationbetween the pointed markers and the segmented markers is then computed.This initial rigid transformation serves to accomplish the transfer fromthe pre-operative reference (reference in which the patterns of the 3Dmodel to be registered are represented) to the robot base reference, andto the endoscopic camera reference (using the calibration of theendoscope in this case). This external registration technique isdescribed in particular in the paper by E. Coste-Manière et al “Optimalplanning of robotically assisted heart surgery: Transfer precision inthe operating room, B. Siciliano and P. Dario, eds, Springer Tracts inAdvanced Robotics, Experimental Robotics VIII, Volume 5, Springer(2002), 424-434.

Once the external registration, if any, has been done, the zone wherethe intervention is to take place is precisely identified. To do this,the surgeon S designates at least one portion of the images displayed onthe screens 16 and 17 and at least one attribute representing each imageportion, via the man/machine interface. The designation of a portion iseffected for example by selecting the portion on the image, with amouse. The designation of an attribute is effected either by a voicecommand, or by selection from a list displayed on the screens 16 and 17or on an auxiliary screen. In the example illustrated in FIG. 4, thesquares A1 to A3 mark the points where the surgeon S “clicks” withhis/her mouse to designate three portions of the region that appear tohim/her to be significant. Similarly, in the example illustrated in FIG.5, the square B marks the point where the surgeon S “clicks” withhis/her mouse to designate the portion of the region that appears tohim/her to be significant.

The attributes are standard information (or classes) each of whichdescribes a known local characteristic or a known local configuration,or in a general manner any information enabling the processing module 22of the device D to determine the matching 3D pattern in the memory 21.

Once in possession of designations supplied by the surgeon S, the deviceD transmits them to its processing module 22, so that it determines inthe memory 21 a 3D pattern which appears to it to be representative ofthe designated portion(s), taking account of the designatedattribute(s).

This determination is performed by an extraction module 23 linked to thememory 21. The processing module 22 must then perform the registrationof this pattern (and the whole of the 3D model of which it is part, asthe case may be). This involves determining how to orient and positionthe pattern so that it can be superimposed on the portion designated bythe surgeon, taking account of the angle at which the interventionregion is observed by the endoscopic camera.

The registration is preferably performed by a registration module 24 ofthe processing module 22. Furthermore, this registration is preferablyrigid, by minimization of a criterion constructed from the measurementequations. The measurement equations are derived in this instance fromhypotheses supplied by the surgeon, when he/she thinks he/she recognizesa portion, such as an artery or a bifurcation between known arteries forexample, or are determined directly by the registration module 24. Ofcourse, another type of registration can be envisaged, particularlyaffine registration.

For example, rigid registration involves determining at least onemeasurement equation from the designation of an image portion, anattribute and at least one hypothesis on the identity of the patterndetermined by the extraction module 23. This registration may also takeinto account the 3D model of the heart when this is stored in the memory21.

Several measurement equations can be determined from the designation ofseveral image portions, several attributes and one or more hypotheses,which constitute a corresponding set of constraints. In this case, theregistration module 24 generates several sets of equations correspondingto the different possible hypotheses and then optimizes the registrationparameters by minimizing a criterion. It then classifies theregistrations obtained as a function of their relevance and selects thebest registration. The parameter estimating method is described inparticular in the paper by T. Vieville et al “Implementing a multi-modelestimation method”, The International Journal of Computer Vision, 44(2001), 41-64.

However, any other known parameter estimation technique can be used.

Once the rigid registration of the 3D pattern is done, it is possible toregister the whole of the 3D model of which said pattern forms a part.Thus, it is the whole 3D model, viewed at the observation angle of theendoscopic camera, and not only the determined and registered 3Dpattern, which can be superimposed onto the displayed or observed image.Of course, certain portions of the 3D model may not be visible due tothe observation angle.

The processing module 22 can then transmit to the control module 15 theposition data of the registered 3D pattern (or of the entire 3D model ofwhich it forms part), in the (calibration) reference of the imagedisplayed (or observed, when the surgeon is equipped with observationspectacles), and the image data which define this 3D pattern (or model),thereby instructing the display device 4 to display it in a superimposedfashion on the observation images delivered by the endoscopic camera 12(or observed through the spectacles).

An overlay of an extensive 3D pattern on an image of the heart isdepicted in FIG. 6. The squares D1 and D2 mark the places where thesurgeon S has clicked with the mouse to designate two portions of theintervention region which appear to him/her to be significant, and theportion of the coronary network superimposed on the image of the heart Hrepresents the 3D pattern registered by the device D.

By virtue of this overlay, the surgeon knows immediately how he/she ispositioned relative to the area that is to be operated on.

In certain situations, the designations made by the surgeon S may notenable the extraction module 23 to determine the 3D pattern whichcorresponds to the selected portion, or to enable the registrationmodule 24 to perform a suitable registration, thereby producing anincorrect overlay of the 3D pattern (or model) on the displayed image.Therefore, the processing module 22 can be arranged so as to determineprecisely the position of the observed region in the reference bysuccessive registrations each based on the extraction of a new 3Dpattern following the designation of at least one other observationimage portion, and at least one attribute representing this otherportion.

In this situation, two cases can be envisaged. In a first case, the 3Dpattern determined by the extraction module 23 is not superimposed onthe image of the observed region in the portion selected by the surgeonS, or the registered 3D pattern is superimposable but does notcorrespond to the structure observed by the surgeon on the selectedportion. The surgeon must then make a new designation to cause theregistration operation to converge. In a second case, it is the controlmodule 15 that automatically notices the error and which sends a messageto the surgeon requesting fresh designations.

On receipt of these new designations, the registration device D repeatsthe processing described above, taking into account the new and olddesignations. In particular, when set up for this purpose, it cancompute new hypotheses and measurement equations.

It can also be envisaged to store in the memory 21 a correspondencetable between the 3D patterns and information data representing them.This may include for example the pattern name, such as for example thename of an artery and/or its branches if any, or the coordinates of atarget point, or operating data recorded in the planning phase, such asthe identification of a stenosis or a calcification zone for example.

In this case, the processing module 22 can be configured so as todeliver the information data associated with a 3D pattern (registered ornot) when it receives the instruction to do so from the control module15 (for example in case of a request by the surgeon S). However, anautomatic mode of operation can also be envisaged wherein, each time theprocessing module 22 registers a 3D pattern and is preparing to deliverthe related position and image data to the control module 15, itextracts the associated information data from the memory 21 so as tocommunicate it substantially simultaneously.

The control module 15 can be also designed, each time it receives arequest designating a 3D pattern of the region observed, so as to, onone hand, instruct the registration device D to determine position datarepresenting the position of this 3D pattern in relation to thecalibration reference, taking account of the registration, and on theother hand, to determine control instructions intended to move themanipulator arms 8 and 9 of the surgical robot 10 to the vicinity of theportion of the region matching the designated 3D pattern.

The installation described above can be used for other types ofoperation, such as open liver or breast surgery for example. In ageneral manner, the installation according to the invention serves toguide the user in the accomplishment of his/her task in an environmentwhere he/she has only a partial and/or distorted view, and/or in adifficult environment.

Furthermore, as previously indicated, the registration device Daccording to the invention can be used in applications other than thosedescribed above. It can be used in particular in installations designedsolely for location guidance (without the intervention of aremotely-operated robot), particularly in urban areas.

It is thus possible to envisage an installation mounted in a roadvehicle, and including one or more cameras delivering real images of theenvironment to a control module for display on screens installed in thedriver's compartment, and a registration device D linked to said controlmodule. The installation can be also connected to an on-board satelliteguidance device, of the GPS type.

In this application, the registration device D stores in its memorypatterns representing the environment (or region) in which the vehiclecan move and determined from digital recordings made previously. Thesepatterns, of which the positions are defined in a selected reference,are for example building facades, or notable buildings or sites, orstatues, or works of art. The memory can also include a volumetric modelof the environment, and a correspondence table between the patterns anddata information on these patterns.

Once the cameras have been calibrated against a calibration reference,the installation can be used.

In this case, when the control module receives a request from apassenger in the vehicle designating a pattern of the observed region(selected from a list of stored patterns using a mouse or by pressing atouch-sensitive display screen), it instructs the registration device Dto determine position data representing the position of this pattern inrelation to the calibration reference, taking into account theadjustment resulting from discrepancies between the selected referenceindex and the calibration index, then to determine control instructionsintended to guide the driver of the vehicle to the portion of the regioncorresponding to this pattern. As indicated above, the control modulemay rely on vehicle position data delivered by the GPS device todetermine the control instructions, as the case may be. Once the vehiclehas arrived at the designated location, the pattern can be superimposedonto the real image of the portion of the region, and information dataassociated with said pattern can be delivered to the passenger.

However, an automatic mode of operation can also be envisaged in whichthe registration device D receives from a passenger in the vehicle thedesignations of at least one observed and displayed portion of a region(selected using a mouse or by pressing the touch-sensitive displayscreen) and at least one attribute representing this portion, such asfor example the nature of a building (house, school, city hall, museum,church) or a location (garden, park, square) or an object (statue,sculpture, work of art), and determines in its memory the pattern whichbest matches this designated portion, taking account of the designatedattribute and the selected viewing angle. It then overlays thedetermined pattern onto the designated image portion. It then derivesthe position data of the registered pattern from the displayed imagereference (calibration reference), and transmits this data together withimage data defining this pattern and any associated information data tothe control module which issues instructions for the data to bedisplayed on the screens, in a superimposed fashion, on the observationimages delivered by the cameras. This mode of operation is thereforesimilar to that described previously in the surgical application.

A variant of the installation can also be envisaged that is adapted forlocation guidance of users traveling on foot in an environment, forexample urban. In this case, the device according to the invention isadvantageously mounted in communications equipment, such as a mobiletelephone for example, equipped with a location function, for example bytriangulation or by GPS, and a camera, and an inertial reference systemas the case may be.

The processing module 22 of the registration device D and the controlmodule 15 of the installations can be made in the form of electroniccircuits, software (or data processing) modules, or a combination ofsoftware modules and electronic circuits.

The invention also relates to a method for real-time registration ofknown patterns, characteristic of a region, onto images of this region.

The method can be implemented with the aid of the registration deviceand the installations described above. Given that the principal andoptional functions and sub-functions provided by the steps of thismethod are substantially identical to those provided by the variousmeans constituting the device and the installations, only the stepsembodying the principal functions of the method according to theinvention will be summarized below.

This method is characterized in that it involves:

-   -   storing in a memory 21 patterns representing portions of a        selected region and of known position and orientation relative        to a common reference,    -   observing the selected region at a selected angle and delivering        observation images of this region in real time,    -   designating at least one portion of the observation image of the        selected region and at least one attribute representing this        portion,    -   identifying in the memory a pattern representing the designated        portion, taking account of the designated attribute, and    -   overlaying the pattern on the designated image portion taking        the selected angle into account.

The invention is not limited to the embodiments of the device,installations and method described above by way of example only, butencompasses all variants that can be envisaged by the person skilled inthe art pursuant to the claims that follow.

1-37. (canceled)
 38. An apparatus comprising: a memory storingpredetermined data corresponding to at least one object, suchpredetermined data comprising an identity of the corresponding object;and a processor configured to: receive an image of a viewed object takenat an angle; accept input from a user, said input comprising adesignation of a portion of the image corresponding to the viewedobject, and a designation of an identity of the viewed object; registerthe predetermined data corresponding to the object having the designatedidentity on the image; and overlay a visual representation of the dataon the image based on the angle.
 39. The apparatus of claim 38, theimage being received from an endoscope.
 40. The apparatus of claim 39,the apparatus further comprising a robotic arm maneuverable within afield of view of the endoscope.
 41. The apparatus of claim 40, theprocessor configured to register the predetermined data on the imagethrough external registration.
 42. The apparatus of claim 41, theexternal registration comprising successively positioning the roboticarm at predetermined locations to align a reference frame of thepredetermined data with a reference frame of the image.
 43. Theapparatus of claim 38, the predetermined data comprising athree-dimensional model representing the corresponding object.
 44. Theapparatus of claim 38, the designation of the portion of the imagecomprising selecting a portion of the image with a mouse.
 45. Theapparatus of claim 38, the designation of the identity comprisingissuing a voice command.
 46. The apparatus of claim 38, the designationof the identity comprising selection from a list displayed on a screen.47. The apparatus of claim 38, the processor configured to register thepredetermined data by minimizing a criterion constructed from at leastone measurement equation.
 48. The apparatus of claim 47, at least onemeasurement equation derived from a hypothesis associated with theviewed object.
 49. The apparatus of claim 38, said input comprisingdesignating multiple portions of the image, and designatingcorresponding identities of viewed objects in the multiple portions. 50.The apparatus of claim 49, the processor configured to register thepredetermined data by multiple successive registrations of the multipledesignated portions.
 51. The apparatus of claim 38, the processorconfigured to register the predetermined data through affineregistration.
 52. The apparatus of claim 38, the visual representationof the data comprising a three-dimensional model of the viewed object.53. The apparatus of claim 40, the processor further configured todetermine control instructions to move the robotic arm to the vicinityof the designated portion of the image.
 54. The apparatus of claim 38,the predetermined data comprising patterns derived from digitalrecordings.
 55. The apparatus of claim 54, the image being received froma camera mounted on a moving vehicle.
 56. The apparatus of claim 38, theidentity of the viewed object comprising at least one attributedescribing a known local characteristic or the viewed object.